Skip to content

[core] Align write validation and file metadata with Java - #1172

Merged
JingsongLi merged 9 commits into
apache:mainfrom
JingsongLi:codex/native-custom-io-auth-partitions
Oct 19, 2026
Merged

JingsongLi merged 9 commits into
apache:mainfrom
JingsongLi:codex/native-custom-io-auth-partitions

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Align Native write validation, binary partition names and data-file creation timestamps with current Java semantics for existing Python read/write/update operations.

Companion Python/Java PR: apache/paimon#10660.

Brief change log

  • Resolve configured or explicit row kinds and apply RowKindFilter before inspecting nested payload values. Keep schema/layout validation and Java's row-kind-specific top-level NOT NULL checks before filtering.
  • Preserve actual NULLs in write-only PyArrow C Data transport, including sliced batches with nonnullable root fields. Core owns the logical row-kind-specific NULL checks; input metadata and nested fields remain intact.
  • Preserve LargeUtf8 rowkind inputs, automatic/precomputed bucket writes, and the generated system-column schema. Ignored ARRAY/ROW/MAP child NULLs no longer fail a valid batch; surviving invalid children still fail.
  • Use stable UTF-8 BINARY/VARBINARY partition values in both legacy-name modes, coordinated with the Java fix. Preserve other legacy types.
  • Persist local wall-clock TIMESTAMP(3) creation fields, as Java's normal DataFileMeta writer factories do.
  • Resolve local creation fields using system ZoneRules when generating REST partition statistics, and reuse that conversion for partition listing. Preserve exact milliseconds and Java DST gap/overlap behavior.
  • Preserve epoch zero and negative creation times in partition statistics; merge multiple file times using Java's maximum regardless of input order.
  • Assign append/PK creation times after auxiliary file indexes and managed Blob references finish.
  • Finish postpone files in arrival order and assign creation timestamps at result/close time, matching Java's replay ordering. Remove the background-close and file-open timestamp machinery.
  • Keep the existing REST authorization implementation and Python bindings. The proposed custom query-authorizer API, callable bridge and related tests have been removed, along with the earlier general Python FileIO bridge.

Tests

  • filtered_write_validation_test: 4 passed, including 24 positive layout/kind/bucket combinations and schema/nullability/INPUT-changelog negative regressions.
  • New filtered-write regressions first reproduce the failure on the old implementation. Cover ARRAY/ROW/MAP, configured/explicit row kinds, LargeUtf8, automatic/precomputed buckets, all-filtered inputs, schema failures, retained invalid payloads, top-level NULL rules and INPUT changelogs.
  • Rust core library: 4307 passed, 6 ignored; creation_time_write_test: 5 passed; postpone_native_write_test: 5 passed.
  • Actual append, PK, input-changelog, postpone and dedicated-Blob writes, commit, manifest roundtrip and applicable reads in Asia/Shanghai and America/New_York.
  • Regressions first fail with the old premature index-output timestamps and epoch-zero/negative statistics.
  • Exact creation-time wire values, epoch zero/negative partition maxima in both file orders, Java DST gap/overlap goldens and postpone close-time/rolling-order regressions. The independent clock oracle honors explicit TZ overrides on Windows.
  • Existing Python-binding Rust unit suite, including Blob callback behavior and real PyArrow C Data write imports: 101 passed. Review follow-up also verifies ordinary STRING and LargeString rowkind inputs against DELETE, UPDATE_BEFORE, INSERT and NULL-key rules through the real WriteState path (8 combinations).
  • Rust 1.98.0 core and binding all-target Clippy with -D warnings, rustfmt and diff checks.
  • Companion Python and Java regressions. Local Python tests did not rebuild or install the Native extension.

API and Format

No Python binding or query-authorization API changes remain. Data-file creation timestamps retain their serialized TIMESTAMP(3) representation; wall-clock/epoch interpretation now matches Java. Binary legacy names use the existing UTF-8 cast instead of allocation identity text.

@JingsongLi JingsongLi changed the title [core] Support resolved FileIO and custom table query authorization [core] Support custom query authorization and align mutation metadata Oct 19, 2026
@JingsongLi JingsongLi changed the title [core] Support custom query authorization and align mutation metadata [core] Align binary partitions and file timestamps with Java Oct 19, 2026
@JingsongLi JingsongLi changed the title [core] Align binary partitions and file timestamps with Java [core] Align write validation and file metadata with Java Oct 19, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 3ae85d3 into apache:main Oct 19, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants